Apparatus, system, and method for adaptive-rate shifting of streaming content

ABSTRACT

An apparatus for adaptive-rate shifting of streaming content includes an agent controller module configured to simultaneously request at least portions of a plurality of streamlets. The agent controller module is further configured to continuously monitor streamlet requests and subsequent responses, and accordingly request higher or lower quality streamlets. A staging module is configured to stage the streamlets and arrange the streamlets for playback on a content player. A system includes a data communications network, a content server coupled to the data communications network and having a content module configured to process content and generate a plurality of high and low quality streams, and the apparatus. A method includes simultaneously requesting at least portions of a plurality of streamlets, continuously monitoring streamlet requests and subsequent responses, and accordingly requesting higher or lower quality streamlets, and staging the streamlets and arranging the streamlets for playback on a content player.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims benefit of U.S. Provisional Patent ApplicationNo. 60/566,831 entitled “APPARATUS, SYSTEM, AND METHOD FOR DYNAMIC RATESHIFTING OF STREAMING CONTENT” and filed on Apr. 30, 2004 for R. DrewMajor and Mark B. Hurst, which is incorporated herein by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The invention relates to video streaming over packet switched networkssuch as the Internet, and more particularly relates to adaptive-rateshifting of streaming content over such networks.

Description of the Related Art

The Internet is fast becoming a preferred method for distributing mediafiles to end users. It is currently possible to download music or videoto computers, cell phones, or practically any network capable device.Many portable media players are equipped with network connections andenabled to play music or videos. The music or video files (hereinafter“media files”) can be stored locally on the media player or computer, orstreamed or downloaded from a server.

“Streaming media” refers to technology that delivers content at a ratesufficient for presenting the media to a user in real time as the datais received. The data may be stored in memory temporarily until playedand then subsequently deleted. The user has the immediate satisfactionof viewing the requested content without waiting for the media file tocompletely download. Unfortunately, the audio/video quality that can bereceived for real time presentation is constrained by the availablebandwidth of the user's network connection. Streaming may be used todeliver content on demand (previously recorded) or from live broadcasts.

Alternatively, media files may be downloaded and stored on persistentstorage devices, such as hard drives or optical storage, for laterpresentation. Downloading complete media files can take large amounts oftime depending on the network connection. Once downloaded, however, thecontent can be viewed repeatedly anytime or anywhere. Media filesprepared for downloading usually are encoded with a higher qualityaudio/video than can be delivered in real time. Users generally dislikethis option, as they tend to want to see or hear the media fileinstantaneously.

Streaming offers the advantage of immediate access to the content butcurrently sacrifices quality compared with downloading a file of thesame content. Streaming also provides the opportunity for a user toselect different content for viewing on an ad hoc basis, whiledownloading is by definition restricted to receiving a specific contentselection in its entirety or not at all. Downloading also supportsrewind, fast forward, and direct seek operations, while streaming isunable to fully support these functions. Streaming is also vulnerable tonetwork failures or congestion.

Another technology, known as “progressive downloads,” attempts tocombine the strengths of the above two technologies. When a progressivedownload is initiated, the media file download begins, and the mediaplayer waits to begin playback until there is enough of the filedownloaded that playback can begin with the hope that the remainder ofthe file will be completely downloaded before playback “catches up.”This waiting period before playback can be substantial depending onnetwork conditions, and therefore is not a complete or fully acceptablesolution to the problem of media presentation over a network.

Generally, three basic challenges exist with regard to data transportstreaming over a network such as the Internet that has a varying amountof data loss. The first challenge is reliability. Most streamingsolutions use a TCP connection, or “virtual circuit,” for transmittingdata. A TCP connection provides a guaranteed delivery mechanism so thatdata sent from one endpoint will be delivered to the destination, evenif portions are lost and retransmitted. A break in the continuity of aTCP connection can have serious consequences when the data must bedelivered in real-time. When a network adapter detects delays or lossesin a TCP connection, the adapter “backs off” from transmission attemptsfor a moment and then slowly resumes the original transmission pace.This behavior is an attempt to alleviate the perceived congestion. Sucha slowdown is detrimental to the viewing or listening experience of theuser and therefore is not acceptable.

The second challenge to data transport is efficiency. Efficiency refersto how well the user's available bandwidth is used for delivery of thecontent stream. This measure is directly related to the reliability ofthe TCP connection. When the TCP connection is suffering reliabilityproblems, a loss of bandwidth utilization results. The measure ofefficiency sometimes varies suddenly, and can greatly impact the viewingexperience.

The third challenge is latency. Latency is the time measure form theclient's point-of-view, of the interval between when a request is issuedand the response data begins to arrive. This value is affected by thenetwork connection's reliability and efficiency, and the processing timerequired by the origin to prepare the response. A busy or overloadedserver, for example, will take more time to process a request. As wellas affecting the start time of a particular request, latency has asignificant impact on the network throughput of TCP.

From the foregoing discussion, it should be apparent that a need existsfor an apparatus, system, and method that alleviate the problems ofreliability, efficiency, and latency. Additionally, such an apparatus,system, and method would offer instantaneous viewing along with theability to fast forward, rewind, direct seek, and browse multiplestreams. Beneficially, such an apparatus, system, and method wouldutilize multiple connections between a source and destination,requesting varying bitrate streams depending upon network conditions.

SUMMARY OF THE INVENTION

The present invention has been developed in response to the presentstate of the art, and in particular, in response to the problems andneeds in the art that have not yet been fully solved by currentlyavailable content streaming systems. Accordingly, the present inventionhas been developed to provide an apparatus, system, and method foradaptive-rate content streaming that overcome many or all of theabove-discussed shortcomings in the art.

The apparatus for adaptive-rate content streaming is provided with alogic unit containing a plurality of modules configured to functionallyexecute the necessary steps. These modules in the described embodimentsinclude an agent controller module configured to simultaneously requesta plurality of streamlets, the agent controller module furtherconfigured to continuously monitor streamlet requests and subsequentresponses, and accordingly request higher or lower quality streamlets,and a staging module configured to stage the streamlets and arrange thestreamlets for playback on a content player.

The apparatus is further configured, in one embodiment, to establishmultiple Transmission Control Protocol (TCP) connections with a contentserver, and request streamlets of varying bitrates. Each streamlet mayfurther comprise a portion of a content file. Additionally, the agentcontroller module may be configured to generate a performance factoraccording to responses from streamlet requests.

In a further embodiment, the agent controller module is configured toupshift to a higher quality streamlet when the performance factor isgreater than a threshold, and the agent controller module determines thehigher quality playback can be sustained according to a combination offactors. The factors may include an amount of contiguously availablestreamlets stored in the staging module, a minimum safety margin, and acurrent read ahead margin.

The agent controller module may be configured to downshift to a lowerquality streamlet when the performance factor is less than a secondthreshold. Also, the agent controller module is further configured toanticipate streamlet requests and pre-request streamlets to enablefast-forward, skip randomly, and rewind functionality. In oneembodiment, the agent controller module is configured to initiallyrequest low quality streamlets to enable instant playback of the contentfile, and subsequent upshifting according to the performance factor.

A system of the present invention is also presented to adaptive-ratecontent streaming. In particular, the system, in one embodiment,includes a data communications network, and a content server coupled tothe data communications network and having a content module configuredto process content and generate a plurality of high and low qualitystreams. In one embodiment, each of the high and low quality streams mayinclude a plurality of streamlets.

In a further embodiment, the system also includes an agent controllermodule configured to simultaneously request a plurality of streamlets,the agent controller module further configured to continuously monitorstreamlet requests and subsequent responses, and accordingly requesthigher or lower quality streamlets, and a staging module configured tostage the streamlets and arrange the streamlets for playback on acontent player.

A method of the present invention is also presented for adaptive-ratecontent streaming. The method in the disclosed embodiments substantiallyincludes the steps necessary to carry out the functions presented abovewith respect to the operation of the described apparatus and system. Inone embodiment, the method includes simultaneously requesting aplurality of streamlets, continuously monitoring streamlet requests andsubsequent responses, and accordingly requesting higher or lower qualitystreamlets, and staging the streamlets and arranging the streamlets forplayback on a content player.

In a further embodiment, the method may include establishing multipleTransmission Control Protocol (TCP) connections with a content server,and requesting streamlets of varying bitrates. Also, the method mayinclude generating a performance factor according to responses fromstreamlet requests, upshifting to a higher quality streamlet when theperformance factor is greater than a threshold, and determining if thehigher quality playback can be sustained. Furthermore, the method mayinclude downshifting to a lower quality streamlet when the performancefactor is less than a second threshold.

In one embodiment, the method includes anticipating streamlet requestsand pre-requesting streamlets to enable fast-forward, skip randomly, andrewind functionality. The method may also comprise initially requestinglow quality streamlets to enable instant playback of a content file, andsubsequent upshifting according to the performance factor.

Reference throughout this specification to features, advantages, orsimilar language does not imply that all of the features and advantagesthat may be realized with the present invention should be or are in anysingle embodiment of the invention. Rather, language referring to thefeatures and advantages is understood to mean that a specific feature,advantage, or characteristic described in connection with an embodimentis included in at least one embodiment of the present invention. Thus,discussion of the features and advantages, and similar language,throughout this specification may, but do not necessarily, refer to thesame embodiment.

Furthermore, the described features, advantages, and characteristics ofthe invention may be combined in any suitable manner in one or moreembodiments. One skilled in the relevant art will recognize that theinvention may be practiced without one or more of the specific featuresor advantages of a particular embodiment. In other instances, additionalfeatures and advantages may be recognized in certain embodiments thatmay not be present in all embodiments of the invention.

These features and advantages of the present invention will become morefully apparent from the following description and appended claims, ormay be learned by the practice of the invention as set forthhereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the advantages of the invention will be readilyunderstood, a more particular description of the invention brieflydescribed above will be rendered by reference to specific embodimentsthat are illustrated in the appended drawings. Understanding that thesedrawings depict only typical embodiments of the invention and are nottherefore to be considered to be limiting of its scope, the inventionwill be described and explained with additional specificity and detailthrough the use of the accompanying drawings, in which:

FIG. 1 is a schematic block diagram illustrating one embodiment of asystem for adaptive rate shifting of streaming content in accordancewith the present invention;

FIG. 2a is a schematic block diagram graphically illustrating oneembodiment of a content file in accordance with the present invention;

FIG. 2b is a schematic block diagram illustrating one embodiment of aplurality of streams having varying degrees of quality and bandwidth inaccordance with the present invention;

FIG. 2c is a schematic block diagram illustrating one embodiment of astream divided into a plurality of streamlets in accordance with thepresent invention;

FIG. 3 is a schematic block diagram illustrating one embodiment of acontent module in accordance with the present invention;

FIG. 4 is a schematic block diagram graphically illustrating oneembodiment of a client module in accordance with the present invention;

FIG. 5 is a schematic flow chart diagram illustrating one embodiment ofa method for processing content in accordance with the presentinvention;

FIG. 6 is a schematic flow chart diagram illustrating one embodiment ofa method for playback of a plurality of streamlets in accordance withthe present invention; and

FIG. 7 is a schematic flow chart diagram illustrating one embodiment ofa method for requesting streamlets within an adaptive-rate contentstreaming environment in accordance with the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Many of the functional units described in this specification have beenlabeled as modules, in order to more particularly emphasize theirimplementation independence. For example, a module may be implemented asa hardware circuit comprising custom VLSI circuits or gate arrays,off-the-shelf semiconductors such as logic chips, transistors, or otherdiscrete components. A module may also be implemented in programmablehardware devices such as field programmable gate arrays, programmablearray logic, programmable logic devices or the like.

Modules may also be implemented in software for execution by varioustypes of processors. An identified module of executable code may, forinstance, comprise one or more physical or logical blocks of computerinstructions which may, for instance, be organized as an object,procedure, or function. Nevertheless, the executables of an identifiedmodule need not be physically located together, but may comprisedisparate instructions stored in different locations which, when joinedlogically together, comprise the module and achieve the stated purposefor the module.

Indeed, a module of executable code may be a single instruction, or manyinstructions, and may even be distributed over several different codesegments, among different programs, and across several memory devices.Similarly, operational data may be identified and illustrated hereinwithin modules, and may be embodied in any suitable form and organizedwithin any suitable type of data structure. The operational data may becollected as a single data set, or may be distributed over differentlocations including over different storage devices, and may exist, atleast partially, merely as electronic signals on a system or network.

Reference throughout this specification to “one embodiment,” “anembodiment,” or similar language means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment of the present invention. Thus,appearances of the phrases “in one embodiment,” “in an embodiment,” andsimilar language throughout this specification may, but do notnecessarily, all refer to the same embodiment.

Reference to a signal bearing medium may take any form capable ofgenerating a signal, causing a signal to be generated, or causingexecution of a program of machine-readable instructions on a digitalprocessing apparatus. A signal bearing medium may be embodied by atransmission line, a compact disk, digital-video disk, a magnetic tape,a Bernoulli drive, a magnetic disk, a punch card, flash memory,integrated circuits, or other digital processing apparatus memorydevice.

Furthermore, the described features, structures, or characteristics ofthe invention may be combined in any suitable manner in one or moreembodiments. In the following description, numerous specific details areprovided, such as examples of programming, software modules, userselections, network transactions, database queries, database structures,hardware modules, hardware circuits, hardware chips, etc., to provide athorough understanding of embodiments of the invention. One skilled inthe relevant art will recognize, however, that the invention may bepracticed without one or more of the specific details, or with othermethods, components, materials, and so forth. In other instances,well-known structures, materials, or operations are not shown ordescribed in detail to avoid obscuring aspects of the invention.

FIG. 1 is a schematic block diagram illustrating one embodiment of asystem 100 for dynamic rate shifting of streaming content in accordancewith the present invention. In one embodiment, the system 100 comprisesa content server 102 and an end user 104. The content server 102 and theend user station 104 may be coupled by a data communications network.The data communications network may include the Internet 106 andconnections 108 to the Internet 106. Alternatively, the content server102 and the end user 104 may be located on a common local area network,wireless area network, cellular network, virtual local area network, orthe like. The end user station 104 may comprise a personal computer(PC), an entertainment system configured to communicate over a network,or a portable electronic device configured to present content.

In the depicted embodiment, the system 100 also includes a publisher110, and a web server 116. The publisher 110 may be a creator ordistributor of content. For example, if the content to be streamed werea broadcast of a television program, the publisher 110 may be atelevision or cable network channel such as NBC®, or MTV®. Content maybe transferred over the Internet 106 to the content server 102, wherethe content is received by a content module 112. The content module 112may be configured to receive, process, and store content. In oneembodiment, processed content is accessed by a client module 114configured to play the content on the end user station 104. In a furtherembodiment, the client module 114 is configured to receive differentportions of a content stream from a plurality of locationssimultaneously. For example, the client module 114 may request andreceive content from any of the plurality of web servers 116.

FIG. 2a is a schematic block diagram graphically illustrating oneembodiment of a content file 200. In one embodiment, the content file200 is distributed by the publisher 110. The content file 200 maycomprise a television broadcast, sports event, movie, music, concert,etc. The content file 200 may also be live or archived content. Thecontent file 200 may comprise uncompressed video and audio, oralternatively, video or audio. Additionally, the content file 200 may becompressed. Examples of a compressed content file 200 include, but arenot limited to, DivX®, Windows Media Video 9®, Quicktime 6.5 Sorenson3®, or Quicktime 6.5/MPEG-4® encoded content.

FIG. 2b is a schematic block diagram illustrating one embodiment of aplurality of streams 202 having varying degrees of quality andbandwidth. In one embodiment, the plurality of streams 202 comprises alow quality stream 204, a medium quality stream 206, and a high qualitystream 208. Each of the streams 204, 206, 208 is a copy of the contentfile 200 encoded and compressed to varying bit rates. For example, thelow quality stream 204 may be encoded and compressed to a bit rate of100 kilobits per second (kbps), the medium quality stream 206 may beencoded and compressed to a bit rate of 200 kbps, and the high qualitystream 208 may be encoded and compressed to 600 kbps.

FIG. 2c is a schematic block diagram illustrating one embodiment of astream 210 divided into a plurality of streamlets 212. As used herein,streamlet refers to any sized portion of the content file 200. Eachstreamlet 212 may comprise a portion of the content contained in stream210, encapsulated as an independent media object. The content in astreamlet 212 may have a unique time index in relation to the beginningof the content contained in stream 210. In one embodiment, the contentcontained in each streamlet 212 has a duration of two seconds. Forexample, streamlet 0 may have a time index of 00:00 representing thebeginning of content playback, and streamlet 1 may have a time index of00:02, and so on. Alternatively, the time duration of the streamlets 212may be any duration smaller than the entire playback duration of thecontent in stream 210. In a further embodiment, the streamlets 212 maybe divided according to file size instead of a time index.

FIG. 3 is a schematic block diagram illustrating in greater detail oneembodiment of the content module 112 in accordance with the presentinvention. The content module 112 may comprise a stream module 302, astreamlet module 304, an encoder module 306, a streamlet database 308,and the web server 116. In one embodiment, the stream module 302 isconfigured to receive the content file 200 from the publisher 110 andgenerate the plurality of streams 202 of varying qualities. The originalcontent file 200 from the publisher may be digital in form and maycomprise content having a high bit rate such as, for example, 2 mbps.The content may be transferred from the publisher 110 to the contentmodule 112 over the Internet 106. Such transfers of data are well knownin the art and do not require further discussion herein. Alternatively,the content may comprise a captured broadcast.

In the depicted embodiment, the plurality of streams 202 may comprisethe low quality stream 204, the medium quality stream 206, and the highquality stream 208. Alternatively, the plurality of streams 202 maycomprise any number of streams deemed necessary to accommodate end userbandwidth. The streamlet module 304 may be configured to receive theplurality of streams 202 from the stream module and generate a pluralityof streams 312, each stream comprising a plurality of streamlets 212. Asdescribed with reference to FIG. 2 c, each streamlet 212 may comprise apre-defined portion of the stream. The encoder module 306 is configuredto encode each streamlet from the plurality of streams 312 and store thestreamlets in the streamlet database 308. The encoding module 306 mayutilize encoding schemes such as DivX®, Windows Media Video 9®,Quicktime 6.5 Sorenson 3®, or Quicktime 6.5/MPEG-4®. Alternatively, acustom encoding scheme may be employed.

The content module 112 may also include a metadata module 312 and ametadata database 314. In one embodiment, metadata comprises staticsearchable content information. For example, metadata includes, but isnot limited to, air date of the content, title, actresses, actors,length, and episode name. Metadata is generated by the publisher 110,and may be configured to define an end user environment. In oneembodiment, the publisher 100 may define an end user navigationalenvironment for the content including menus, thumbnails, sidebars,advertising, etc. Additionally, the publisher 110 may define functionssuch as fast forward, rewind, pause, and play that may be used with thecontent file 200. The metadata module 312 is configured to receive themetadata from the publisher 110 and store the metadata in the metadatadatabase 314. In a further embodiment, the metadata module 312 isconfigured to interface with the client module 114, allowing the clientmodule 114 to search for content based upon at least one of a pluralityof metadata criteria. Additionally, metadata may be generated by thecontent module 112 through automated process(es) or manual definition.

Once the streamlets 212 have been received and processed, the clientmodule 114 may request stream lets 212 using HTTP from the web server116. Such use of client side initiated requests requires no additionalconfiguration of firewalls. Additionally, since the client module 114initiates the request, the web server 116 is only required to retrieveand serve the requested streamlet. In a further embodiment, the clientmodule 114 may be configured to retrieve streamlets 212 from a pluralityof web servers 310. Each web server 116 may be located in variouslocations across the Internet 106. The streamlets 212 are essentiallystatic files. As such, no specialized media server or server-sideintelligence is required for a client module 114 to retrieve streamlets212. Streamlets 212 may be served by the web server 116 or cached bycache servers of Internet Service Providers (ISPs), or any other networkinfrastructure operators, and served by the cache server. Use of cacheservers is well known to those skilled in the art, and will not bediscussed further herein. Thus, a highly scalable solution is providedthat is not hindered by massive amounts of client module 114 requests tothe web server 116 at any specific location.

FIG. 4 is a schematic block diagram graphically illustrating oneembodiment of a client module 114 in accordance with the presentinvention. The client module 114 may comprise an agent controller module402, a streamlet cache module 404, and a network controller module 406.In one embodiment, the agent controller module 402 is configured tointerface with a viewer 408, and transmit streamlets 212 to the viewer408. In a further embodiment, the client module 114 may comprise aplurality of agent controller modules 402. Each agent controller module402 may be configured to interface with one viewer 408. Alternatively,the agent controller module 402 may be configured to interface with aplurality of viewers 408. The viewer 408 may be a media player (notshown) operating on a PC or handheld electronic device.

The agent controller module 402 is configured to select a quality levelof streamlets to transmit to the viewer 408. The agent controller module402 requests lower or higher quality streams based upon continuousobservation of time intervals between successive receive times of eachrequested streamlet. The method of requesting higher or lower qualitystreams will be discussed in greater detail below with reference to FIG.7.

The agent controller module 402 may be configured to receive usercommands from the viewer 408. Such commands may include play, fastforward, rewind, pause, and stop. In one embodiment, the agentcontroller module 402 requests streamlets 212 from the streamlet cachemodule 404 and arranges the received streamlets 212 in a staging module409. The staging module 409 may be configured to arrange the streamlcts212 in order of ascending playback time. In the depicted embodiment, thestreamlets 212 are numbered 0, 1, 2, 3, 4, etc. However, each streamlet212 may be identified with a unique filename.

Additionally, the agent controller module 402 may be configured toanticipate streamlet 212 requests and pre-request streamlets 212. Bypre-requesting streamlets 212, the user may fast-forward, skip randomly,or rewind through the content and experience no buffering delay. In afurther embodiment, the agent controller module 402 may request thestreamlets 212 that correspond to time index intervals of 30 secondswithin the total play time of the content. Alternatively, the agentcontroller module 402 may request streamlets at any interval less thanthe length of the time index. This enables a “fast-start” capabilitywith no buffering wait when starting or fast-forwarding through contentfile 200. In a further embodiment, the agent controller module 402 maybe configured to pre-request streamlets 212 corresponding to specifiedindex points within the content or within other content in anticipationof the end user 104 selecting new content to view.

In one embodiment, the streamlet cache module 404 is configured toreceive streamlet 212 requests from the agent controller module 402.Upon receiving a request, the streamlet cache module 404 first checks astreamlet cache 410 to verify if the streamlet 212 is present. In afurther embodiment, the streamlet cache module 404 handles streamlet 212requests from a plurality of agent controller modules 402.Alternatively, a streamlet cache module 404 may be provided for eachagent controller module 402. If the requested streamlet 212 is notpresent in the streamlet cache 410, the request is passed to the networkcontroller module 406. In order to enable fast forward and rewindcapabilities, the streamlet cache module 404 is configured to store theplurality of streamlets 212 in the streamlet cache 410 for a specifiedtime period after the streamlet 212 has been viewed. However, once thestreamlets 212 have been deleted, they may be requested again from theweb server 116.

The network controller module 406 may be configured to receive streamletrequests from the streamlet cache module 404 and open a connection tothe web server 116 or other remote streamlet 212 database (not shown).In one embodiment, the network controller module 406 opens a TCP/IPconnection to the web server 116 and generates a standard HTTP GETrequest for the requested streamlet 212. Upon receiving the requestedstreamlet 212, the network controller module 406 passes the streamlet212 to the streamlet cache module 404 where it is stored in thestreamlet cache 410. In a further embodiment, the network controllermodule 406 is configured to process and request a plurality ofstreamlets 212 simultaneously. The network controller module 406 mayalso be configured to request a plurality of streamlets, where eachstreamlet 212 is subsequently requested in multiple parts.

In a further embodiment, streamlet requests may comprise requestingpieces of any streamlet file. Splitting the streamlet 212 into smallerpieces or portions beneficially allows for an increased efficiencypotential, and also eliminates problems associated with multiplefull-streamlet requests sharing the bandwidth at any given moment. Thisis achieved by using parallel TCP/IP connections for pieces of thestreamlets 212. Consequently, efficiency and network loss problems areovercome, and the streamlets arrive with more useful and predictabletiming.

In one embodiment, the client module 114 is configured to use multipleTCP connections between the client module 114 and the web server 116 orweb cache. The intervention of a cache may be transparent to the clientor configured by the client as a forward cache. By requesting more thanone streamlet 212 at a time in a manner referred to as “parallelretrieval,” or more than one part of a streamlet 212 at a time,efficiency is raised significantly and latency is virtually eliminated.In a further embodiment, the client module allows a maximum of threeoutstanding streamlet 212 requests. The client module 114 may maintainadditional open TCP connections as spares to be available should anotherconnection fail. Streamlet 212 requests are rotated among all openconnections to keep the TCP flow logic for any particular connectionfrom falling into a slow-start or close mode. If the network controllermodule 406 has requested a streamlet 212 in multiple parts, with eachpart requested on mutually independent TCP/IP connections, the networkcontroller module 406 reassembles the parts to present a completestreamlet 212 for use by all other components of the client module 114.

When a TCP connection fails completely, a new request may be sent on adifferent connection for the same streamlet 212. In a furtherembodiment, if a request is not being satisfied in a timely manner, aredundant request may be sent on a different connection for the samestreamlet 212. If the first streamlet request's response arrives beforethe redundant request response, the redundant request can be aborted. Ifthe redundant request response arrives before the first requestresponse, the first request may be aborted.

Several streamlet 212 requests may be sent on a single TCP connection,and the responses are caused to flow back in matching order along thesame connection. This eliminates all but the first request latency.Because multiple responses are always being transmitted, the processinglatency of each new streamlet 212 response after the first is not afactor in performance. This technique is known in the industry as“pipelining.” Pipelining offers efficiency in request-responseprocessing by eliminating most of the effects of request latency.However, pipelining has serious vulnerabilities. Transmission delaysaffect all of the responses. If the single TCP connection fails, all ofthe outstanding requests and responses are lost. Pipelining causes aserial dependency between the requests.

Multiple TCP connections may be opened between the client module 114 andthe web server 116 to achieve the latency-reduction efficiency benefitsof pipelining while maintaining the independence of each streamlet 212request. Several streamlet 212 requests may be sent concurrently, witheach request being sent on a mutually distinct TCP connection. Thistechnique is labeled “virtual pipelining” and is an innovation of thepresent invention. Multiple responses may be in transit concurrently,assuring that communication bandwidth between the client module 114 andthe web server 116 is always being utilized. Virtual pipeliningeliminates the vulnerabilities of traditional pipelining. A delay in orcomplete failure of one response does not affect the transmission ofother responses because each response occupies an independent TCPconnection. Any transmission bandwidth not in use by one of multipleresponses (whether due to delays or TCP connection failure) may beutilized by other outstanding responses.

A single streamlet 212 request may be issued for an entire streamlet212, or multiple requests may be issued, each for a different part orportion of the streamlet. If the streamlet is requested in severalparts, the parts may be recombined by the client module 114 streamlet.

In order to maintain a proper balance between maximized bandwidthutilization and response time, the issuance of new streamlet requestsmust be timed such that the web server 116 does not transmit theresponse before the client module 114 has fully received a response toone of the previously outstanding streamlet requests. For example, ifthree streamlet 212 requests are outstanding, the client module 114should issue the next request slightly before one of the three responsesis fully received and “out of the pipe.” In other words, request timingis adjusted to keep three responses in transit. Sharing of bandwidthamong four responses diminishes the net response time of the other threeresponses. The timing adjustment may be calculated dynamically byobservation, and the request timing adjusted accordingly to maintain theproper balance of efficiency and response times.

The schematic flow chart diagrams that follow are generally set forth aslogical flow chart diagrams. As such, the depicted order and labeledsteps are indicative of one embodiment of the presented method. Othersteps and methods may be conceived that are equivalent in function,logic, or effect to one or more steps, or portions thereof, of theillustrated method. Additionally, the format and symbols employed areprovided to explain the logical steps of the method and are understoodnot to limit the scope of the method. Although various arrow types andline types may be employed in the flow chart diagrams, they areunderstood not to limit the scope of the corresponding method. Indeed,some arrows or other connectors may be used to indicate only the logicalflow of the method. For instance, an arrow may indicate a waiting ormonitoring period of unspecified duration between enumerated steps ofthe depicted method. Additionally, the order in which a particularmethod occurs may or may not strictly adhere to the order of thecorresponding steps shown.

FIG. 5 is a schematic flow chart diagram illustrating one embodiment ofa method 500 for processing content in accordance with the presentinvention. In one embodiment the method 500 starts 502, and the contentmodule 112 receives 504 content from the publisher 110. Receivingcontent 504 may comprise receiving 504 a digital copy of the contentfile 200, or digitizing a physical copy of the content file 200.Alternatively, receiving 504 content may comprise capturing a radio ortelevision broadcast. Once received 504, the stream module 302 generates506 a plurality of streams 202, each stream 202 having a differentquality. The quality may be predefined, or automatically set accordingto end user bandwidth, or in response to pre-designated publisherguidelines

The streamlet module 304 receives the streams 202 and generates 508 aplurality of streamlets 212. In one embodiment, generating 508streamlets comprises dividing the stream 202 into a plurality of twosecond streamlets 212. Alternatively, the streamlets may have any lengthless than or equal to the length of the stream 202. The encoder module306 then encodes 510 the streamlets according to a compressionalgorithm. In a further embodiment, the algorithm comprises aproprietary codec such as WMV9®. The encoder module 306 then stores 512the encoded streamlets in the streamlet database 308. Once stored 512,the web server 116 may then serve 514 the streamlets. In one embodiment,serving 514 the streamlets comprises receiving streamlet requests fromthe client module 114, retrieving the requested streamlet from thestreamlet database 308, and subsequently transmitting the streamlet tothe client module 114. The method 500 then ends 516.

FIG. 6 is a schematic flow chart diagram illustrating one embodiment ofa method 600 for viewing a plurality of streamlets in accordance withthe present invention. The method 600 starts and an agent control module402 is provided 604 and associated with a viewer 408 and provided with astaging module 409. The agent controller module 402 then requests 606 astreamlet from the streamlet cache module 404. Alternatively, the agentcontroller module 402 may simultaneously request 606 a plurality ofstreamlets from the streamlet cache module 404. If the streamlet isstored 608 locally in the streamlet cache 410, the streamlet cachemodule 404 retrieves 610 the streamlet and sends the streamlet to theagent controller module 402. Upon retrieving 610 or receiving astreamlet, the agent controller module 402 makes 611 a determination ofwhether or not to shift to a higher or lower quality stream 202. Thisdetermination will be described below in greater detail with referenceto FIG. 7.

In one embodiment, the staging module 409 then arranges 612 thestreamlets into the proper order, and the agent controller module 402delivers 614 the streamlets to the viewer 408. In a further embodiment,delivering 614 streamlets to the end user comprises playing video and oraudio streamlets on the viewer 408. If the streamlets are not stored 608locally, the streamlet request is passed to the network controllermodule 406. The network controller module 406 then requests 616 thestreamlet from the web server 116. Once the streamlet is received, thenetwork controller module 406 passes the streamlet to the streamletcache module 404. The streamlet cache module 404 archives 618 thestreamlet. Alternatively, the streamlet cache module 404 then archives618 the streamlet and passes the streamlet to the agent controllermodule 402, and the method 600 then continues from operation 610 asdescribed above.

Referring now to FIG. 7, shown therein is a schematic flow chart diagramillustrating one embodiment of a method 700 for requesting streamletswithin a adaptive-rate shifting content streaming environment inaccordance with the present invention. The method 700 may be used in oneembodiment as the operation 611 of FIG. 6. The method 700 starts and theagent controller module 402 receives 704 a streamlet as described abovewith reference to FIG. 6. The agent controller module 402 then monitors706 the receive time of the requested streamlet. In one embodiment, theagent controller module 402 monitors the time intervals A betweensuccessive receive times for each streamlet response. Ordering of theresponses in relation to the order of their corresponding requests isnot relevant.

Because network behavioral characteristics fluctuate, sometimes quitesuddenly, any given A may vary substantially from another. In order tocompensate for this fluctuation, the agent controller module 402calculates 708 a performance ratio r across a window of n samples forstreamlets of playback length S. In one embodiment, the performanceratio r is calculated using the equation

$r = {S{\frac{n}{\sum\limits_{i = 1}^{n}\Delta_{i}}.}}$

Due to multiple simultaneous streamlet processing, and in order tobetter judge the central tendency of the performance ratio r, the agentcontrol module 402 may calculate a geometric mean, or alternatively anequivalent averaging algorithm, across a window of size m, and obtain aperformance factor φ:

$\phi_{current} = {( {\prod\limits_{j = 1}^{m}r_{j}} )^{\frac{1}{m}}.}$

The policy determination about whether or not to upshift 710 playbackquality begins by comparing φ_(current) with a trigger threshold Θ_(up).If φ_(current)≥Θ_(up), then an up shift to the next higher qualitystream may be considered 716. In one embodiment, the trigger thresholdΘ_(up) is determined by a combination of factors relating to the currentread ahead margin (i.e. the amount of contiguously available streamletsthat have been sequentially arranged by the staging module 409 forpresentation at the current playback time index), and a minimum safetymargin. In one embodiment, the minimum safety margin may be 24 seconds.The smaller the read ahead margin, the larger Θ_(up) is to discourageupshifting until a larger read ahead margin may be established towithstand network disruptions. If the agent controller module 402 isable to sustain 716 upshift quality, then the agent controller module402 will upshift 717 the quality and subsequently request higher qualitystreams. The determination of whether use of the higher quality streamis sustainable 716 is made by comparing an estimate of the higherquality stream's performance factor, φ_(higher), with Θ_(up). Ifφ_(higher)≥Θ_(up) then use of the higher quality stream is consideredsustainable. If the decision of whether or not the higher stream rate issustainable 716 is “no,” the agent control module 402 will not attemptto upshift 717 stream quality. If the end of the stream has been reached714, the method 618 ends 716.

If the decision on whether or not to attempt upshift 710 is “no”, adecision about whether or not to downshift 712 is made. In oneembodiment, a trigger threshold Θ_(down) is defined in a manneranalogous to Θ_(up). If φ_(current)>Θ_(down) then the stream quality maybe adequate, and the agent controller module 402 does not downshift 718stream quality. However, if φ_(current)≤Θ_(down), the agent controllermodule 402 does downshift 718 the stream quality. If the end of thestream has not been reached 714, the agent controller module 402 beginsto request and receive 704 lower quality streamlets and the method 618starts again. Of course, the above described equations and algorithmsare illustrative only, and may be replaced by alternative streamletmonitoring solutions.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the invention is, therefore, indicatedby the appended claims rather than by the foregoing description. Allchanges which come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

What is claimed is:
 1. An apparatus for rendering a video that isadaptively received as a digital stream from a video server over anetwork, the apparatus comprising: a media player operating on theapparatus, wherein the media player is configured to stream the videofrom the video server via at least one transmission control protocol(TCP) connection over the network, wherein the video server storesmultiple different copies of the video encoded at different bit rates asmultiple sets of streamlets, wherein each of the streamlets yields adifferent portion of the video on playback, wherein the streamletsacross the different copies yield the same portions of the video onplayback, and wherein the streamlets in the different copies are alignedin time such that the streamlets that play back the same portion of thevideo for the different copies each begin at the same playback time inrelation to the beginning of the video, and wherein the media playerstreams the video by: requesting sequential streamlets of one of thecopies from the video server according to the playback times of thestreamlets, wherein the sequential streamlets are selected by the mediaplayer from the based upon successive determinations to shift theplayback quality to a higher or lower quality one of the differentcopies of the video; adapting the successive determinations to shift theplayback quality to achieve continuous playback of the video using thestreamlets of the highest quality copy of the video that is determinedto be sustainable at that time; and presenting the video for playback byproviding the requested streamlets in order of ascending start time. 2.The apparatus of claim 1 wherein the media player repeatedly generates afactor relating to the performance of the network that is indicative ofan ability to sustain the streaming of the video, and wherein theadapting of the successive determinations occurs based upon the factor.3. The apparatus of claim 1, wherein each streamlet is a portion of acontent file maintained by the video server that provides one of thecopies of the video.
 4. The apparatus of claim 1, wherein the requestingthe sequential streamlets comprises the apparatus transmitting hypertexttransport protocol (HTTP) GET requests for selected streamlets, whereineach of the HTTP GET requests identifies a separate portion of a filestored by the video server, wherein the portion of the file correspondsto the requested streamlet.
 5. The apparatus of claim 1 wherein each ofthe streamlets of each of the different copies is independentlyrequestable and playable by the apparatus.
 6. The apparatus of claim 4,wherein the requesting of the sequential streamlets comprises the enduser device transmitting hypertext transport protocol (HTTP) GETrequests for selected streamlets.
 7. The apparatus of claim 1 whereineach of the streamlets in each of the plurality of different copies is aseparate file stored by the video server.
 8. The apparatus of claim 1wherein the media player upshifts to a higher quality one of thedifferent copies when the factor is greater than a first threshold anddownshifts to a lower quality one of the different copies when thefactor is less than a second threshold.
 9. The apparatus of claim 1,wherein the requesting the sequential streamlets comprises the apparatustransmitting hypertext transport protocol (HTTP) GET requests forselected streamlets, and wherein each of the HTTP GET requestsidentifies a separately-identifiable portion of the one or more filesthat corresponds to the requested streamlet.
 10. The apparatus of claim1, wherein the apparatus is configured to initially request low qualitystreamlets to enable instant playback of the content file, andsubsequent upshifting according to the performance factor.
 11. The enduser device of claim 1 wherein each of the streamlets in each of theplurality of different copies is a separately-identifiable portion ofone or more files stored by the video server.
 12. The apparatus of claim1 wherein the apparatus is a mobile computing device comprising aprocessor and a non-transitory data storage.
 13. The apparatus of claim1, wherein the apparatus is configured to establish multipleTransmission Control Protocol (TCP) connections with a content server,and to request streamlets of varying bitrates from the content servervia the multiple TCP connections.
 14. A method executable by an end userdevice to stream a video received via a connection with a server over anetwork, the method comprising: requesting, by the end user device, aplurality of sequential streamlets of one of the copies from the serverbased on playback times of the streamlets wherein multiple differentcopies of the video encoded at different bit rates are stored asmultiple sets of streamlets on the server, wherein each of thestreamlets yields a different portion of the video on playback, whereinthe streamlets across the different copies yield the same portions ofthe video on playback, and wherein each of the streamlets comprises aplayback time such that each of the streamlets for each of the differentcopies that encode the same portion of the video begins at the sameplayback time in relation to the beginning of the video; makingsuccessive determinations by the end user device to shift the playbackquality to achieve continuous playback of the video using the streamletsof the highest quality copy determined sustainable at that time; andpresenting the video by playing back the requested media streamlets onthe end user device in order of ascending playback time.
 15. The methodof claim 14 wherein the making of the successive determinations to shiftcomprises upshifting to a higher quality one of the different copieswhen the at least one factor is greater than a first threshold anddownshifting to a lower quality one of the different copies when the atleast one factor is less than a second threshold.
 16. The method ofclaim 15 wherein each of the streamlets of each of the different copiesis independently requestable and playable by the end user device. 17.The method of claim 14 wherein each of the streamlets in each of theplurality of different copies is a separately-identifiable portion ofone or more files stored by the video server.
 18. The method of claim 17wherein the requesting of the sequential streamlets comprises the enduser device transmitting hypertext transport protocol (HTTP) GETrequests for selected streamlets, and wherein each of the HTTP GETrequests identifies the separately-identifiable portion of the one ormore files that corresponds to the requested streamlet.
 19. The methodof claim 14, wherein the video captures a live event, and wherein thestreamlets of the different copies are available to the end user devicewhile the live event is occurring.
 20. The method of claim 14 furthercomprising the end user device repeatedly generating a factor that isindicative of an ability to sustain the streaming of the video, andwherein the successive determinations to shift the playback quality aremade based upon the factor.